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(57) A more efficient data structure and method for 
browsing a multimedia is disclosed. In the present in- 
vention, link information between segments and seman- 
tic elements of a multimedia data structure is generated 
with minimal use of storage. Also, priority or weight of 
segments is represented based on semantic elements 
and priority or weight of semantic elements is represent- 
ed based on segments to allow a search of multimedia 
based on content. The priority or weight information is 
represented as an attribute of the link information to in- 
crease the efficiency of the storage and a method of au- 
tomatically extracting the priority or weight information 
is disclosed. 
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Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 



[0001] The present invention relates to a multimedia 
browser, and more particularly to a structure and meth- 
od for browsing the multimedia with defined priorities 
and/or weights of multimedia data segments and se- 
mantic elements. 

Description of the Related Art 

[0002] Contents of a multimedia program are typically 
divided into a structural content and a semantic content. 
[0003] The structural content represents how a multi- 
media program is physically and logically organized. For 
example, for a news program, a structural content may 
be organized by an initial title portion, a headline news 
portion, and a main news portion including news items. 
For a drama or a movie, a structural content may be or- 
ganized according to video sections of the drama or 
movie story as developed with respect to time. Here, the 
headline news portion of a news program deals with a 
plurality of news items and may be further divided into 
sections corresponding to each news item. Similarly, the 
main news portion may further be divided into sections 
corresponding to news items and each news item may 
be divided into anchor shots and reporter shots. 
[0004] As described above, division of a multimedia 
program into several portions is called segmentation 
and the divided portions are called segments. That is, 
the structural content is generally described with seg- 
ments. In the above example, if the news program is 
logically divided, the news program would have a tree 
structure representing an example of the structural con- 
tent. 

[0005] Fig. 1 shows an example structure information 
of an audio/video stream. As shown in Fig. 1 , the audio/ 
video stream may be represented by a group of scene 
segments, and the scene segments may be described 
with a tree structure of shot segments. The structure in- 
formation of segments is not fixed to a single structure 
but can have various other structure according to differ- 
ent viewpoints or criteria for the segmentation. Namely, 
different segments would result from segmentation with 
reference to audio information, visual information, or 
logical information. However, any such information is 
called structure content or structure information and 
each segment is called a structure element. 
[0006] On the other hand, the semantic content is in- 
formation representing human perception and an ob- 
ject-oriented description of a multimedia program. The 
semantic content may include abstract concepts such 
as fear, sadness, and happiness; objects such as char- 
acters, places, or things appearing in the multimedia 
program; events which occur in the multimedia program; 
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and relation connecting the objects and events. For ex- 
ample, in a drama, the actors, the character roles, and 
events describing a plot would be the semantic content; 
and each actor, character role, and event would be the 

5 semantic elements. If the semantic elements are de- 
fined by relations among the elements, a semantic 
structure would be generated. 
[0007] Fig. 2 is an example semantic information for 
a motion picture such as a drama or a movie. As shown, 

io the story of Pinocchio is represented by a semantic ori- 
ented description with characters such as Geppetto and 
Pinocchio, and relations and roles between Geppetto 
and Pinocchio. 

[0008] The semantic elements.and structure informa- 
15 tion as described above has relations. The relation may 
be represented by 'a semantic element described in a 
segment of a selected program' or in reverse, 'a seg- 
ment including a particular semantic element.' There are 
other various methods and structures for describing the 
20 relations, depending on the way the semantic elements 
and the structural information are formed. For the ex- 
amples shown in Figs. 1 and 2, there may be information 
indicating the segments in which the character Pinoc- 
chio appears or there may be information on a segment 
25 indicating the objects and events shown in the segment 
such as Pinocchio. 

[0009] Using the above link information, i.e. the rela- 
tions, users may easily browse a multimedia section or 
content as desired. Since a multimedia data includes the 

30 link information of the structure information and seman- 
tic information, users can search for a specific segment 
of a video or audio to allow a more convenient browsing. 
That is, users can specifically browse portions of a video 
rather than watching the whole video, and effectively un- 

35 derstand the contents of the whole video in a short pe- 
riod of time by browsing a summary of the video content. 
[0010] For such browsing, each semantic element in- 
cludes information connecting the semantic elements to 
each corresponding segment, or each segment in- 

40 eludes information connecting the segment to each cor- 
responding semantic element. Moreover, both are or- 
ganized into a data structure including a priority or a key 
word for each segment or semantic element. According- 
ly, a multimedia has video segments and semantic ele- 

45 ments which allow browsing by connecting semantic el- 
ement information based on video segments or by con- 
necting video segments based on semantic elements. 
[0011] Furthermore, a video is divided into units of a 
shot and multimedia browsing may be carried out based 

so upon a shot segmentation and a shot clustering by 
which each shot is divided and grouped into units of a 
scene. Namely, using the shot information and scene 
information, key frames which represent a scene can be 
set for a video browsing and searching. Therefore, users 

55 may selectively browse or search a specific portion of a 
video stream. For example, a video browsing method 
has been proposed using a hierarchically decomposed 
or a scene transition graph describing a temporal rela- 
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tionship among scenes. 
[0012] As described above, to allow a browsing of a 
specific portion of a video, typical video indexing sys- 
tems establish a data structure based on each semantic 
characteristic element. That is, a browsing based on ap- 
pearances of characters or on event sections is provid- 
ed. However, in such multimedia browser based on con- 
tent, a significant amount of overlap occurs to describe 
each semantic elements of each segments or each seg- 
ments of semantic elements. 

SUMMARY OF THE INVENTION 

[0013] Accordingly, an object of the present invention 
is to solve at least the problems and disadvantages of 
the related art. 

[0014] Therefore, an object of the present invention is 
to provide a data structure and method for a more effi- 
cient browsing of data. 

[0015] Another object of the present invention is to 
provide a data structure and method with reduced over- 
lapping in description of data. 
[0016] A still another object of the present invention 
is to provide a data structure to describe priority and/or 
weight (priority/weight) information based on semantic 
elements which is described with link information be- 
tween structure elements and semantic ielements, and 
a structure for describing the priority/weight information 
in which qualitative information representing how pre- 
cisely a structure element describes a semantic element 
is quantitatively described in the link information. 
[0017] A further object of the present invention is to 
provide a structure for describing priority/weight infor- 
mation based on segments in which information repre- 
senting priority/weight of semantic elements is differen- 
tiated depending on the semantic elements. 
[0018] A still further object of the present invention is 
to provide a structure for defining and describing indi- 
vidual priority/weight of segments and semantic ele- 
ments of a multimedia in which link information includes 
both priority/weight information based on the semantic 
elements and priority/weight information based on the 
segments to describe individual priority/weight for cor- 
responding elements. 

[0019] Additional advantages, objects, and features 
of the invention will be set forth in part in the description 
which follows and in part will become apparent to those 
having ordinary skill in the art upon examination of the 
following or may be learned from practice of the inven- 
tion. The objects and advantages of the invention may 
be realized and attained as particularly pointed out in 
the appended claims. 

[0020] To achieve the objects and in accordance with 
the purposes of the invention, as embodied and broadly 
described herein, a multimedia data structure includes 
link information for connecting a semantic element 
structure and a segment information structure, where 
the link information includes priority/weight information 



between semantic elements and segments as an at- 
tribute, when a multimedia data Includes the semantic 
element structure describing a content of a multimedia 
and the segment information structure of the multimedia 
5 for browsing the multimedia based on content. The pri- 
ority/weight information may further includes informa- 
tion representing how precise the priority/weight is de- 
scribed. 

[0021] In the above embodiment, the priority/weight 

10 information may be priority/weight information based on 
the semantic elements representing the priority/weight 
of semantic information related to a segment. The pri- 
ority/weight information may be priority/weight informa- 
tion of structure elements which varies depending on the 

15 semantic elements. The priority/weight information may 
also be priority information based on semantic ele- 
ments, which represents priority/weight of the segments 
related to a semantic element or priority/weight informa- 
tion of semantic elements which varies depending on 

so the structure elements. 

[0022] Moreover, the priority/weight information may 
be information based on both the semantic elements 
and the segments. In such case, the link information in- 
cluding the priority/weight information may have addi- 

25 tional information describing connection relationships. 
Furthermore, the priority information may be described 
as a segment locator for locating a corresponding seg- 
ment when link information includes the priority/weight 
information based on the semantic elements. The prior- 

30 ity/weight information may also be described as a se- 
mantic element locator for locating a corresponding se- 
mantic element when the link information includes the 
priority/weight information based on the segment. 
[0023] Another embodiment of a multimedia data 

35 structure according to the present invention includes link 
information for connecting a semantic element and a 
segment wherein the link information includes priority/ 
weight information of the structure element which varies 
depending on the semantic element, when a multimedia 

40 data includes the semantic element structure describing 
the contents of a multimedia and the segment informa- 
tion structure of the multimedia for browsing the multi- 
media based on the contents. 

[0024] A still another embodiment of a multimedia da- 
*5 ta structure according to the present invention includes 
link information for connecting a semantic element and 
a segment wherein the link information includes priority/ 
weight information of the semantic element which varies 
depending on the structure element, when a multimedia 
50 data includes the semantic element structure describing 
the contents of a multimedia and the segment informa- 
tion structure of the multimedia for browsing the multi- 
media based on the contents. 

[0025] A further embodiment of a multimedia data 
55 structure according to the present invention includes link 
information for connecting a semantic element and a 
segment wherein the link information includes both pri- 
ority/weight information of the structure element which 
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varies depending on the semantic element and priority/ 
weight information of the semantic element which varies 
depending on the structure element, when a multimedia 
data includes the semantic element structure describing 
the contents of a multimedia and the segment informa- 5 
tion structure of the multimedia for browsing the multi- 
media based on content. 

[0026] A method for browsing a multimedia according 
to the present invention, includes selecting a semantic 
element or segment information, selecting a link infor- 10 
mation of the selected semantic element or segment, 
and displaying corresponding segment or semantic el- 
ement information in sequence according to the priority/ 
weight between the corresponding segment element 
and segment which was obtained by the link informa- *5 
tion, wherein a multimedia data includes a semantic el- 
ement structure describing a content of a multimedia 
and a segment information structure of the multimedia 
for browsing the multimedia based on content. 
[0027] Another method for browsing a multimedia ac- 20 
cording to the present invention, includes assigning 'se- 
mantic element n* as a semantic element, selecting 'se- 
mantic element n' and reading link information of seg- 
ments assigned by 'semantic element n,' determining 
which segments are connected to 'semantic element n' 25 
from the link information and reading priority/weight of 
the connection from the priority information included in 
the corresponding link information, and displaying the 
corresponding segments in sequence according to the 
priority, wherein the priority/weight information based on 30 
the semantic element is described by attributes of link 
information with the corresponding segments. 
[0028] A further embodiment of a method for browsing 
a multimedia according to the present invention includes 
assigning 'segment n' as a structure element, selecting 35 
'segment n' and reading link information of semantic el- 
ements assigned by 'segment n,' determining which se- 
mantic elements are connected to 'segment n' from the 
link information and reading priority/weight of the con- 
nection from the priority/weight information included in *o 
the corresponding link information, and displaying the 
corresponding semantic elements in sequence accord- 
ing to the priority/weight, wherein priority/weight infor- 
mation based on the structure element described by at- 
tributes of link information with the corresponding se- 
mantic elements. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0029] The invention will be described in detail with so 
reference to the following drawings in which like refer- 
ence numerals refer to like elements wherein: 

Fig. 1 is a view showing a schematic structure of 
audio and video information in the related art; 55 
Fig. 2 is a view showing an example of semantic 
information of a multimedia; 
Fig. 3 is a view showing a structure for describing 



priority/weight information based on a semantic 
structure according to the present invention; 
Fig. 4 is a view showing a structure for describing 
priority/weight information based on segments ac- 
cording to another embodiment of the present in- 
vention; and 

Fig. 5 is a view showing a structure for describing 
priority/weight information based on both a seman- 
tic structure and segments according to the present 
invention. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0030] Reference will now be made in detail to the 
present invention, examples of which are illustrated in 
the accompanying drawings. 

[0031] If a multimedia is a motion picture, the contents 
of the multimedia would include structure information of 
segments as shown in Fig. 1 and semantic elements as 
shown in Fig. 2. The structure information and the se- 
mantic elements have relationships and in the present 
invention, the relationships are links from semantic ele- 
ments to segments as shown in Fig. 3 or links from seg- 
ments to semantic elements as shown in Fig. 4. 
[0032] Assuming that a semantic element Pinocchio 
is set, locators which designate segments correspond- 
ing to the semantic element Pinocchio, such as seg- 
ments or segments with scenes in which Pinocchio ap- 
pears, can represent a semantic based link information. 
Similarly, locators which designate semantic elements 
corresponding to a segment, such as objects or con- 
cepts in the segment, can represent a segment based 
link information. Here, each semantic element may have 
additional information independent from the semantic 
based link information. In the above assumption, the ad- 
ditional information may be, for example, the actor who 
represents Pinocchio or the information describing a 
character called Pinocchio. On the other hand, the struc- 
ture information may have information independent 
from the segment based link information such as how 
and where each data segment exists. 
[0033] Also, each semantic element may be de- 
scribed in several segments. In the example, the char- 
acter Pinocchio may be defined without redundancy as 
an object in the semantic structure and described in sev- 
eral portions of a multimedia data. At this time, a 'relation 
locator' is necessary to link the object Pinocchio of the 
semantic structure to portions in which the object is de- 
scribed. Such portions may be segments in which 
Pinocchio appears in a video clip, an audio clip in which 
the voice of Pinocchio is recorded, photographs of 
Pinocchio, or character data related to Pinocchio, and 
can be defined as a structure element. 
[0034] In the linked structure element describing se- 
mantic elements, the degree of relation between the 
structure elements and the semantic elements may be 
different. That is, in some video clips, the back of Pinoc- 
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chio may appear in the backgrouTRTand in other video 
clips, the face of Pinocchio may appear in close-up. 
Such degrees of description or relations will be referred 
as a priority, weight, or importance in this invention. 
[0035] The link information may represent or describe 
the contents of a motion picture with different focus such 
as on characters, events, background, or relationship 
changes among the characters. Moreover, when a mul- 
timedia is searched based on content, semantic based 
links to segments or segment based links to semantic 
elements would have different priorities/weights de- 
pending on a viewpoint of the user or the contents of a 
motion picture. For example, a section of a motion pic- 
ture in which the main character Pinocchio appears may 
be important in understanding the development of a sto- 
ry, but may also be an insignificant detail with respect to 
the story. 

[0036] If the priority/weight or importance of the links 
between the semantic elements and the segments are 
represented by an attribute of the link information, the 
contents of the multimedia may be browsed more effi- 
ciently and conveniently using the priority/weight infor- 
mation. Accordingly, in the present invention, the link in- 
formation between the semantic elements and the seg- 
ments includes priority/weight information of the links as 
an attribute. Fig. 3 is an example structure for describing 
the priority/weight information based on semantic ele- 
ments and Fig. 4 is an example structure for describing 
the priority/weight information based on the structure el- 
ements. 

[0037] The priority/weight information, as shown in 
Figs. 3 and 4, is represented by an attribute of the links 
between the semantic element and the segment. The 
degree of the priority/weight is represented by a number. 
[0038] Referring to Fig. 3, structure elements 301 in- 
clude a plurality of segments and link information 303 
connects corresponding segments to semantic ele- 
ments 302 of a multimedia contents, where the link in- 
formation 303 includes priorities. 
[0039] The priority/weight of the link information 303 
is represented by numbers written on arrows from the 
semantic elements 302 to corresponding segments 
301 . Here, the link information 303 may be represented 
by connectors, relations or locators. Thus, the priority or 
weight of the link information 303 may be one of an at- 
tribute of the link information 303. 
[0040] In the link information 303 of Fig. 3, a semantic 
element Pinocchio connects to segment 1 , segment 6, 
segment 1 0, and segment n+2 of the structure elements 
301 . Here, the priority/weight of the link information 303 
to segment 1 is '0.9,' the priority/weight of the link infor- 
mation 303 to segments 6 and 10 are '0.5; and the pri- 
ority/weight of the link information 303 to segment n+2 
is '0.3.' Assuming that the priority/weight ranges from 0 
to 1 and that the priority/weight is based on sections in 
which Pinocchio appears, the priority/weight of segment 
1 would be highest and the priority/weight of segment 
n+2 would be the lowest. For example, segment 1 may 



therront 



include sections in which therront of Pinocchio appears 
in close-up while segment n+2 includes a video clip in 
which the back of Pinocchio appears in the background 
for a brief moment. 

5 [0041] As described above, by assigning priority/ 
weight information based on the semantic elements as 
an attribute of the link information between semantic el- 
ements and corresponding segments, a more effective 
browsing and searching of a multimedia is allowed for 

10 a user. For example, assume that a user is searching 
the contents of the story Pinocchio based on the seman- 
tic elements for the character Pinocchio. The link infor- 
mation 303 of the segments linked with the semantic el- 
ement Pinocchio would be used to display the sections 

15 in which Pinocchio appears. However, using the priority/ 
weight information of the link information 303, the seg- 
ments of the structure elements 301 which corresponds 
to the semantic element Pinocchio can be displayed, in 
sequence, from the highest to the lowest priority/weight. 

20 [0042] Furthermore, more than one semantic element 
may be described in a segment of the structure informa- 
tion and, depending upon the segment, the priorities/ 
weights of the semantic elements would vary. For ex- 
ample, even though Pinocchio is the most important se- 

25 mantic element in the story of Pinocchio, the appear- 
ance of Pinocchio in the background of a scene in which 
Geppetto is speaking to neighbor is not significant to the 
story. In such segments, Geppetto would be the more 
important semantic element. Thus, it would also be help- 
so ful for a user to know the relative priorities/weights of 
semantic elements such as concepts, objects, or events 
described in each segment of the structure information. 
Such priority/weight information may be represented by 
another attribute of the link information. 

35 [0043] Fig. 4 shows a second embodiment of a struc- 
ture for describing the priority/weight information based 
on the structure elements including structure elements 
401 of a plurality of segments connected to semantic 
elements 402 by link information 403 with priority infor- 

40 mation . As in the first embodiment, the priority/weight of 
the link information 403 is represented by numbers writ- 
ten on arrows from the structure elements 401 to corre- 
sponding semantic elements 402. Here, the link infor- 
mation 403 may be represented by connectors, rela- 

45 tions, or locators. Thus, the priority/weight or weight of 
the link information 403 may be another one of an at- 
tribute of the link information 403. 
[0044] In the link information 403 of Fig. 4, a segment 
n+2 connects to semantic elements Pinocchio, the 

so Cricket Pepe, and the concept of adventure. Here, the 
priority/weight of the link information 403 to Pinocchio 
is '0.3/ the priority/weight of the link information 403 to 
the Cricket Pepe is '0.7/ and the priority/weight of the 
link information 403 to the concept of adventure is '0.8.' 

55 Assuming that the priority/weight ranges from 0 to 1 , 
even though Pinocchio may be the most important se- 
mantic element of the story, the priority/weight of the 
concept of adventure is higher than the priority/weight 
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of Pinocchio for the segment n+2. 
[0045] Therefore, by assigning priority/weight infor- 
mation based on the structure elements as an attribute 
of the link information between segments and corre- 
sponding semantic elements, a more effective browsing 
and searching of a multimedia is allowed for a user. For 
example, assume that a user is searching the contents 
of the story Pinocchio based on the structure elements 
by viewing a segment n. The link Information 403 of the 
semantic elements linked with the segment n would be 
used to display the whole segment. However, using the 
priority/weight information of the link information 403, 
the semantic elements can be displayed in order from 
the highest to the lowest priority/weight, and scenes or 
shots of the segment describing the corresponding se- 
mantic elements may be displayed, in sequence, from 
the highest to the lowest priority. 
[0046] In the above data structures based on either 
semantic elements of Fig. 3 and structure elements of 
Fig. 4, the priority/weight information is included as an 
attribute of the link information between the semantic 
elements and the structure elements, wherein the link 
information describes the priority/weight of either struc- 
ture elements based on the semantic elements or se- 
mantic elements based on the structure elements. Fig. 
5 shows a third embodiment of a structure for describing 
the priority/weight based on both the semantic elements 
and based on the segments, where the priority/weight 
is an attribute of the link information. 
[0047] Referring to Fig. 5, a link ID represents identi- 
fications of the link information between segments and 
semantic elements, wherein link Li connects segment 
Sj with semantic element Mk. For each link Li, the prior- 
ity/weight information includes both priority/weight of 
corresponding segment Sj based on semantic element 
Mk, i.e. semantic based importance, and priority/weight 
of corresponding semantic element Mk based on seg- 
ment Sj, i.e. segment based importance. 
[0048J The value which represents the priority/weight 
may be set to a fixed range of integers or a real numbers, 
or integers or real numbers without a fixed range. In the 
preferred embodiment, the value of the priority is fixed 
between '0' and '1 Accordingly, if there is no change in 
the stored segments, the assigned values of the priority/ 
weight information may be normalized and controlled to 
a number between '0' and '1 ,' regardless of the range of 
priority/weight values assigned. Also, an authorized us- 
er (or a manager) may subjectively assign the priority/ 
weight information, which may be modified by further 
study or input. 

[0049] However, in a system in which the priority/ 
weight values are assigned without a fixed range or if 
there is a continuous change in the stored data, normal- 
ization cannot be achieved or the priority/weight values 
must be normalized each time there is a change in the 
data. In suchcase, one method to set the priority/weight 
information, described by a real number, to a value be- 
tween '0' and T is as follows. 
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[0050] For an image, the priority/weight value may be 
set by a ratio Ip which is the amount of a semantic ele- 
ment that occupies an image [Ip = pixel number of se- 
mantic image/total image pixel number]. For a video clip, 

5 the priority/weight value may be set by a temporal ratio 
Vp which is the length of a section in which a semantic 
element appears [Vp = Ip x (time describing semantic 
element/total segment time)]. For an audio, the priority/ 
weight value may be set by a ratio Ap, a combination of 

10 the degree of sound pressure of a semantic element and 
the length of time in which the sound is heard [Ap = 
(sound pressure of semantic element/total sound pres- 
sure) x (time of sound pressure of semantic element/ 
segment time)]. Thus, the priority/weight information 

15 can automatically be set to a value between '0' and '1 .' 
[0051] As described above, when the priority/weight 
is represented by an attribute of the link information, the 
efficiency of the browsing and storage are increased. 
First, the semantic elements need only be stored once 

20 jn the semantic structure, such that it is not necessary 
to redundantly store the semantic elements. Second, 
when the semantic information is separated from the 
structure information, the link information exists regard- 
less of the priority/weight information. Thus, the priority/ 

25 weight information may be added to the link information 
as attributes using one or two real numbers, thereby re- 
ducing the size of data for describing the priority/weight 
information. 

[0052] Furthermore, when a user wishes to find a vid- 

30 eo clip related to a specific semantic element during a 
search, a search engine may obtain the semantic ele- 
ment from the semantic structure and the link informa- 
tion corresponding to the semantic element from the link 
information. The search engine then delivers segments 

35 designated by the obtained link information together 
with the priority/weight information to the user. There- 
fore, the user may view segments according to the pri- 
ority/weight information which is quantitatively repre- 
sented. Similarly, when a user wishes to find information 

*o jn a specific segment during a search, related semantic 
elements described in the segment may be obtained us- 
ing the link information corresponding to the segment, 
thereby allowing the user to view the semantic elements 
related to the segment. 

45 [0053] In the present invention, when a multimedia 
data has link information between semantic elements 
and structure elements for search of multimedia based 
on content, the link information includes priority/weight 
information as an attribute such that the efficiency of the 

50 search, storage, and browsing of the multimedia are in- 
creased. Moreover, the priority/weight information may 
automatically be extracted and represented, case by 
case, such that indexing of the multimedia may be ef- 
fectively performed. 

55 [0054] The foregoing embodiments are merely exem- 
plary and are not to be construed as limiting the present 
invention. The present teachings can be readily applied 
to other types of apparatuses. The description of the 
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present invention is intended to bCTnustrative, and not 
to limit the scope of the claims. Many alternatives, mod- 
ifications, and variations will be apparent to those skilled 
in the art. 



Claims 

1 . A multimedia data structure comprising: 

a structure information of a plurality of seg- 
ments; 

a semantic information of a plurality of semantic 
elements; 

a link information which connects at least one 
segment to corresponding at least one seman- 
tic element, wherein said link information in- 
cludes, as an attribute, one of at least a priority 
or weight information between each connected 
at least one segment and at least one semantic 
element. 

2. A multimedia data structure of claim 1 , wherein the 
link information is a link from a first semantic ele- 
ment to each segment which describes the first se- 
mantic element. 

3. A multimedia data structure of claim 2, wherein the 
priority or weight information is based on the first 
semantic element and indicates the importance of 
the first semantic element in each of the linked seg- 
ments. 

4. A multimedia data structure of claim 1 , wherein the 
link information is a link from a first segment to each 
semantic element which are described in the first 
segment. 

5. A multimedia data structure of claim 4, wherein the 
priority or weight information is based on the first 
segment and indicates the importance of each 
linked semantic element in the first segment. 

6. A multimedia data structure of claim 1 , wherein the 
link information includes semantic based impor- 
tance of a segment and segment based importance 
of a semantic element between the connected at 
least one segment and at least one semantic ele- 
ment. 

7. A multimedia data structure of claim 1 , wherein the 
link information is represented by one of connec- 
tors, relations, or locators. 

8. A multimedia data structure of claim 1 , wherein the 
priority or weight information is a value ranging from 
0 to 1 , where the value of 1 has the most impor- 
tance. 
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9. A multimedia data strucifflBof claim 8, wherein set- 
ting one of a priority or a weight value for an image 
by a ratio which is an amount of a semantic element 
that occupies the image; setting a corresponding 

s one of a priority or weight value for a video clip by 
a temporal ratio which is the length of a section in 
which a semantic element appears in a segment; 
and setting a corresponding one of a priority or 
weight value for an audio by a ratio which is a com- 

10 bination of the degree of sound due to a semantic 
element and the length of time in which the sound 
is heard during a segment. 

10. A system for browsing multimedia comprising: 

15 

receiving a user selection of either a semantic 
element from semantic information or a seg- 
ment from structure information; 
obtaining link information corresponding to a 

20 selected semantic element or segment, where- 

in the link information includes at least one of a 
priority or weight information as an attribute of 
the link information; and 
displaying segments linked with a selected se- 

25 mantic element in sequence according to the 

priority or weight information using the link in- 
formation, if a semantic element is selected for 
browsing based on semantic elements; and 
displaying sections which describes semantic 

30 elements linked with a selected segment in se- 

quence according to the priority or weight infor- 
mation using the link information, if a segment 
is selected for browsing based on segments. 

35 11. A system of claim 10, wherein the link information 
connects a selected segment to corresponding at 
least one semantic element if a segment is select- 
ed, and wherein the link information connects a se- 
lected semantic element to corresponding at least 

40 one segment if a semantic element is selected. 

12. A system of claim 10, wherein the link information 
is a link from a first semantic element to each seg- 
ment which describes the first semantic element, if 

45 a first semantic element is selected. 

13. A system of claim 12, wherein the priority or weigh 
information is based on the first semantic element 
and indicates the importance of the first semantic 

so element in each of the linked segments. 

14. A system of claim 10, wherein the link information 
is a link from a first segment to each semantic ele- 
ment which are described in the first segment, if a 

55 first segment is selected. 

15. A system of claim 14, wherein the priority or weight 
information is based on the first segment and indi- 
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cates the importance of eachlffraed semantic ele- 
ment in the first segment. 



16. A system of claim 10, wherein the link information 

is represented by one of connectors, relations, or 5 
locators. 

17. A system of claim 10, wherein the priority or weight 
information is a value ranging from 0 to 1 , where the 
value of 1 has the most importance. 10 

18. A method of displaying results of multimedia brows- 
ing comprising: 

storing a structure information of a plurality of 15 
segments, and a semantic information of a plu- 
rality of semantic elements; 
storing link information which links at least one 
segment to corresponding at least one seman- 
tic element, wherein said link information in- 20 
eludes, as an attribute, at least one or a priority 
information or weight Information between the 
connected at least one segment and at least 
one semantic element; 

obtaining link information corresponding to a 25 
semantic element or a segment, selected by a 
user for browsing; and 

displaying segments linked with a selected se- 
mantic element in sequence according to the 
priority or weight information using the link in- 30 
formation, if a semantic element is selected for 
browsing based on semantic elements; and 
displaying sections which describes semantic 
elements linked with a selected segment in se- 
quence according to the priority or weight infor- 35 
mation using the link information, if a segment 
is selected for browsing based on segments. 

19. A method of claim 18, wherein the priority or weight 
information of the link information includes seman- *o 
tic based importance of a segment and segment 
based importance of a semantic element between 

the connected at least one segment and at least one 
semantic element. 

45 

20. A method of claim 18, wherein the link information 
is represented by one of connectors, relations, or 
locators. 
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